Visuomotor Learning Using Image Manifolds

Vision provides us with much information about distal spaces, and enables us to formulate expectations about how objects move. In this work, we consider how images, together with motor and performance data, can be used to form ”maps” or a holistic view of any motion. For example if the learner knows how to throw a ball, it may be able to use the map to see that it also applies to kicking, or catching, or throwing darts. These maps, we suggest, are manifolds that constitute dense latent spaces for these motions. While earlier approaches to learning such systems (e.g. discovering laws of physics) used prior knowledge of the set of system variables, we assume no such knowledge and discover an equivalent representation with alternate parameters (the latent parameters of the non-linear manifold). We show how such a system can be built without any explicit parametrization for a number of motion systems from rolling on an incline to pulleys to projectile motion. We then demonstrate how such a map (for projectiles) can be used for throwing darts, and how practice can improve the relevant part of the map, and hence the throwing capability.